Add general ignition-ostree module with rootfs replacement by cgwalters · Pull Request #184 · coreos/fedora-coreos-config

cgwalters · 2019-09-26T00:14:54Z

Previously the 40coreos-var module was "special /var
handling for Ignition + OSTree".

This new module takes over that, renaming it to ignition-ostree,
and extends it with support for replacing the root filesystem.

cgwalters · 2019-09-26T00:16:51Z

This is work for coreos/fedora-coreos-tracker#94

ajeddeloh · 2019-09-26T18:50:46Z

This is blocked by the rootfs mount options issue, right?

cgwalters · 2019-09-26T18:59:43Z

This is blocked by the rootfs mount options issue, right?

Working on that!

darkmuggle

I like the direction. My only question is whether reading it to an in-ram location (tar -cf ...) might be faster since you avoid the overhead of the delete process. For some cloud-based network attach storage, there's going to be a cost and avoiding that could make it marginally faster.

overlay.d/05core/usr/lib/dracut/modules.d/31ignition-ostree/ignition-ostree-dracut-rootfs.sh

cgwalters · 2019-09-27T15:02:55Z

Will rebase on #187

cgwalters · 2019-09-27T15:03:43Z

My only question is whether reading it to an in-ram location (tar -cf ...) might be faster since you avoid the overhead of the delete process.

Yeah I started with that, but hit tar+SELinux issues. My plan is to put this in rpm-ostree, see coreos/rpm-ostree#1911

overlay.d/05core/usr/lib/dracut/modules.d/31ignition-ostree/ignition-ostree-dracut-rootfs.sh

ajeddeloh · 2019-09-27T20:36:40Z

overlay.d/05core/usr/lib/dracut/modules.d/31ignition-ostree/ignition-ostree-dracut-rootfs.sh

+        fi
+        echo "Restoring rootfs from RAM..."
+        for x in boot ostree; do
+            mv -Tn ${tmproot}/${x} ${rootmnt}/${x}


What's the impact of this and hardlinks? Should we be doing ostree operations instead of just mv/cp?

Yeah, I plan to move this portion into rpm-ostree.

overlay.d/05core/usr/lib/dracut/modules.d/31ignition-ostree/ignition-ostree-dracut-rootfs.sh

This adds basic infrastructure units for "re-provisioning" the root filesystem. See: coreos/fedora-coreos-tracker#94 A unit first detects if the Ignition configuration has a filesystem with the label `root` - if so, we save the rootfs into RAM, let `ignition-disks.service` run, then restore it from RAM. Earlier attempts copied the files into `tmpfs`; this instead uses the `brd` kernel module which is a RAM-backed block device so we can just `dd`. This a faster and more reliable way to save the rootfs. However, `brd` doesn't support discards, so we require at minimum $rootfs_size RAM (e.g. 3G) until reprovisioning is complete. In the future I might investigate trying a `tmpfs` again as that better integrates with the page cache and doesn't have the discard issue. But it requires more work to save the rootfs. Future work here will likely more the `restore` phase into `rpm-ostree`.

- add check for `wipeFilesystem == true` in jq query - use `modprobe --first-time` instead of `lsmod` to avoid race condition - use `rsync` instead of `mv`; they're mostly equivalent in this context, but rsync is more tuned for this sort of massive operation and does things slightly more correctly (specifically, it correctly labels the root inode of the rootfs itself)

jlebon · 2020-05-23T22:00:27Z

As mentioned in coreos/fedora-coreos-tracker#94 (comment), I've now sent the patch upstream to be able to read labels from the initrd. In order to be confident it worked, I rebased this on top of the latest testing-devel. With the kernel patch + this PR, I can boot up FCOS on ext4 with SELinux on and the following Ignition config:

"storage": {
    "filesystems": [
        {
            "device": "/dev/disk/by-partlabel/root",
            "format": "ext4",
            "wipeFilesystem": true,
            "label": "root"
        }
    ]
}

Anyway, took the liberty to force-push the rebase to this PR. I also added a follow-up commit with some tweaks. We can squash that in if you approve.

cgwalters · 2020-05-24T14:06:30Z

overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-dracut-rootfs.sh

+        echo "Moved rootfs to RAM, pending redeployment: ${memdev}"
+        ;;
+    restore)
+        mount "$rootdisk" $rootmnt


Aside: I am still leaning towards "rootfs reprovisioning must add a root= kernel argument" (UUID is a sane default, but it doesn't have to be that). That way we avoid all races with changing the meaning of /dev/disk/by-label/root.

Yeah, I think I agree with this. I feel like we should spend some time to write up some of the complex root device options we want to support (e.g. at least starting with multipath, RAID, and LUKS) and see how it fits into one cohesive picture.

This is done in #503.

jlebon · 2020-06-29T21:47:49Z

So circling back, once we unblock the SELinux patch (I have an RHBZ pending for that), this is the next crucial piece in the puzzle for coreos/fedora-coreos-tracker#94. On its own, it will allow e.g. reconfiguring /dev/sda4 from xfs to ext4 or moving it to another disk entirely. #503 builds on it to support targeting complex devices.

It's been pretty solid for me while hacking on coreos/fedora-coreos-tracker#94. Though we should get some fresh eyes on it too.

jlebon · 2020-07-02T16:51:51Z

This is now part of #503.

* nav: Add link to Afterburn docs * nav: Improve projects docs naming and link names

This was referenced Sep 26, 2019

dracut: Add 31ignition-ostree module coreos/ignition-dracut#112

Closed

WIP: Redeploy rootfs coreos/ignition-dracut#107

Closed

darkmuggle approved these changes Sep 26, 2019

View reviewed changes

overlay.d/05core/usr/lib/dracut/modules.d/31ignition-ostree/ignition-ostree-dracut-rootfs.sh Outdated Show resolved Hide resolved

cgwalters force-pushed the ignition-ostree-rootfs branch from 90f3db1 to 9540b70 Compare September 27, 2019 17:49

ajeddeloh reviewed Sep 27, 2019

View reviewed changes

cgwalters force-pushed the ignition-ostree-rootfs branch 2 times, most recently from fbc96e6 to a5dc2eb Compare December 2, 2019 15:15

cgwalters mentioned this pull request Dec 2, 2019

05core: relabel /var from initrd #245

Merged

cgwalters and others added 2 commits May 23, 2020 16:00

jlebon force-pushed the ignition-ostree-rootfs branch from a5dc2eb to f1cd51b Compare May 23, 2020 21:57

cgwalters commented May 24, 2020

View reviewed changes

This was referenced Jun 26, 2020

Support for reconfiguring the root storage coreos/fedora-coreos-tracker#94

Closed

40ignition-ostree: add coreos-inject-rootmap.service #503

Merged

jlebon closed this Jul 2, 2020

dustymabe pushed a commit to jbtrystram/fedora-coreos-config that referenced this pull request Apr 19, 2024

Add link to Afterburn and improve sidebar link text (coreos#184)

1f086cc

* nav: Add link to Afterburn docs * nav: Improve projects docs naming and link names

Conversation

cgwalters commented Sep 26, 2019

Uh oh!

cgwalters commented Sep 26, 2019

Uh oh!

ajeddeloh commented Sep 26, 2019

Uh oh!

cgwalters commented Sep 26, 2019

Uh oh!

darkmuggle left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cgwalters commented Sep 27, 2019

Uh oh!

cgwalters commented Sep 27, 2019

Uh oh!

Uh oh!

ajeddeloh Sep 27, 2019

Choose a reason for hiding this comment

Uh oh!

cgwalters Dec 2, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jlebon commented May 23, 2020

Uh oh!

cgwalters May 24, 2020

Choose a reason for hiding this comment

Uh oh!

jlebon May 25, 2020

Choose a reason for hiding this comment

Uh oh!

jlebon Jun 29, 2020

Choose a reason for hiding this comment

Uh oh!

jlebon commented Jun 29, 2020

Uh oh!

jlebon commented Jul 2, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants